System and method for monitoring of states of components of a microscope

ABSTRACT

System for state monitoring of a microscope the system having at least one measuring sensor in each case for capturing at least one time-variable chemical and/or physical quantity, a camera for recording an image in a field of view and a processing unit. The at least one measuring sensor has a display area and displays thereon a measured value for the captured time-variable chemical and/or physical quantity. The camera is arranged so that the display areas of at least one measuring sensor are located in the field of view and the processing unit is configured to evaluate the image and to extract the display areas contained in the image therefrom. Also, a method for state monitoring of a microscope is disclosed, wherein at least one measuring sensor with a display area is provided in order to capture in each case at least one time-variable chemical and/or physical quantity, and an image is recorded. The image is recorded so that it contains the display areas of at least one measuring sensor. The display areas are identified in the image, the image is evaluated and the measured values contained in the image are extracted.

The present invention relates to a system for state monitoring of amicroscope, the system having at least one measuring sensor, in eachcase for capturing at least one time-variable chemical and/or physicalquantity, a camera for recording an image in a field of view, and aprocessing unit. Furthermore, the present invention relates to a methodfor state monitoring of a microscope, wherein at least one measuringsensor is provided, which measuring sensor captures in each case atleast one time-variable chemical and/or physical quantity, and an imageis recorded.

Approaches of machine learning are known, for example, from thepublications Bishop, “Pattern Recognition and Machine Learning”,Springer, 2006; Krizhevsky et al.: “ImageNet Classification with DeepConvolutional Neural Networks”, NIPS, 2012; Razavian et al.: “CNNFeatures off-the-shelf: an Astounding Baseline for Recognition”,CVPR-Workshops, 2014; Shelhamer et al.: “Fully Convolutional Networksfor Semantic Segmentation”, IEEE Trans. Pattern Anal, Mach, Intel,39(4): pp. 640-651, 2017 and Girshick et al.: “Rich feature hierarchiesfor accurate object detection and semantic segmentation”, CVPR, 2014.

Based thereof, a flexible and at the same time simplified monitoring ofstates of a plurality of optical components of a microscope should beenabled.

The invention is defined in claims 1 and 12 defined. Advantageousfurther developments are set forth in the dependent claims. Thepreferred embodiments apply in the same way to the system and themethod.

A system for state monitoring of a microscope is provided. The systemhas at least one measuring sensor, for capturing in each case at leastone time-variable chemical and/or physical quantity. The at least onemeasuring sensor has a display area and displays thereon a measuredvalue for the captured time-variable chemical and/or physical quantity.The system further has a camera for recording an image in a field ofview and a processing unit. The camera is arranged so that the displayarea of at least one measuring sensor is located in the field of view,and the processing unit is configured to evaluate the image and toextract therefrom the measured values displayed on the display areascontained in the image.

At least one measuring sensor is provided in a method for statemonitoring of a microscope, said at least one measuring sensor capturingat least one time-variable chemical and/or physical quantity. The atleast one measuring sensor has a display area on which a measured valuefor the at least one time-variable chemical and/or physical quantity isdisplayed. An image is recorded. The image is recorded so that itcontains the display areas of at least one measuring sensor. The imageis evaluated and the measured values displayed on the display areascontained in the image are extracted.

The measuring sensors measure at least one time-variable chemical and/orphysical quantity of optical or mechanical components in the microscopeor its surroundings and display the measured value either directly or inthe form of values or data derived therefrom. In this case, the capturedtime-variable chemical and/or physical quantity can apply directly tothe component or have a direct effect on it. For example, informationcan be output on the display areas as “ready”, “suboptimal”, or “notusable”. The measuring sensors can be attached to all components of themicroscope, such as, for example, to the lens and/or to the stage, butalso on a sample carrier, an incubator or the like. Also, an attachmentof measuring sensors associated with one or more optical or mechanicalcomponents, for example for measuring a temperature, a humidity, abrightness or a vibration, is possible. Preferably, the temperature, aCO2 content, an O2 content, a pressure, a pH value, a humidity, anexposure to light and/or a filling level is measured at or near thecomponent. Also, the measurement of other time-variable chemical and/orphysical quantities is possible.

For example, thermochromic test strips, pH test strips, etc. are used asmeasuring sensors. In principle, the user is free to choose themeasurement sensors; however, the measurement sensor used must displaythe value or the derived value or the data in an externally identifiablemanner on a display area. The measured value can be displayed directlyon the display areas, but a coded display, e.g. as a scale, color code,or in grayscale is also possible. This is achieved by designing themeasuring sensor as a thermochromic element, or by providing themeasuring sensor with an analog or digital display area. Thermochromicelements change their color when the temperature changes.

The system can also be used for fluorescence microscopes.

The camera, which is provided in the system, has a field of view fromwhich it records an image. A camera, which is already present on themicroscope, can also be used; a motor-controlled pivotable camera, thatis to say motor-controlled image adjustment, is also conceivable. Thecamera is aligned with the display areas of the measurement sensors insuch a way that the display area(s) of the at least one measurementsensor lies/lie in the field of view and/or the pivoting range. Itrecords an image of the display area(s) once, continuously or atspecific times. The type and nature of the camera are irrelevant as longas the display areas and thus the information to be captured arecontained in the image. Several cameras are also possible; as far asreference is made to a camera below, this is purely exemplary.

In the image, however, not only the relevant information of the displayareas is contained, but also other elements, such as, for example, partsof the microscope, which are not of interest for monitoring because theydo not have any display areas that would have to be evaluated. Toevaluate the image recorded by the camera, the processing unit isconfigured in such a way that it identifies the display areas containedin the image by identifying them and extracting the area itself or thedisplayed value. Optionally, the display areas contained in the imagecan be identified locally and used to support the evaluation.

In a preferred embodiment, the processing unit or the method isconfigured in such a way that the display areas are filtered from theimage and joined together to form an overall image, for example bystitching. In this way, the measurement results of the measurementsensors are visible at a glance.

The filtering out of the display areas can be supported manually in aninitial setup by a learning step in which the display areas in the imageare manually defined. In this case, it is possible for the user, forexample, to specify the location of each display area in the imageand/or the component relevant for the display area.

In a further preferred embodiment, the processing unit or the method isconfigured to read out the measured values from the display areas of themeasuring sensors by means of text and/or image analysis. For example,the image analysis is divided in steps. First, the display areas in theimage are identified and found by the processing unit. Subsequently,optionally, the type of each measuring sensor (manufacturer, model,etc.) is identified based on the image. In a further step, the measuredvalue displayed on the display area of the measuring sensor isidentified and assigned to a chemical and/or physical quantity and/orcomponent.

A data connection between the measuring sensors and the processing unitcan preferably be dispensed with both in the system and in the method.The camera only needs to be connected to the processing unit for imagedata exchange, which means that cost-effective measuring sensors can beused and the measuring sensors can be used flexibly on differentcomponents on the microscope or in proximity of the components of themicroscope, and assembly can be cost-effective. In addition, the optionof retrofitting an existing system is much easier to implement.

Since the measuring sensors in the system only have to show the measuredvalue on their display area, very simple measuring sensors can be used.Preferably, the measuring sensors have an internal energy source and donot need any supply connections. In a preferred embodiment of thesystem, the measuring sensors are formed as passive sensors, that is tosay sensors that do not require any electrical energy, or as moduleswith an internal energy supply, such as, for example, battery-operatedmodules and/or modules with a rechargeable battery. Compared tocontrolled probes or measuring sensors, which would have to be speciallydesigned for the respective measured quantity and the measuringlocation, these measuring sensors are extremely inexpensive and ensureconsiderable cost savings.

In a preferred embodiment of the system, the processing unit has amemory in which measured values are provided with a time stamp and arestored. In a preferred embodiment of the method, the measured values areprovided with a time stamp and stored. In doing so, the temporalevolution of measured values can be traced accurately. In addition, thecurrent measured values can be compared at any time with the samemeasured values at other times.

In a preferred embodiment of the system or the method, a display deviceis provided on which the extracted display areas and/or measured valuesare displayed. In this case, the display device is preferably formed asa warning device by means of which the system, controlled by theprocessing unit, signals an overshooting or undershooting of fixedmeasured values to the user.

It is also preferred if the results of the extraction of the measuredvalues displayed in the display areas are transmitted to a control loopand/or a workflow, where they are used to control a process, to readjusta process and/or to carry out the workflow. For example, the measuredvalues can be used to automatically readjust the temperature in theexamination setup.

In a preferred embodiment an illumination source is provided, whichilluminates several of the display area(s) of the at least one measuringsensor, preferably all. Illumination of the display area(s) increasesthe visibility of the measured values which are displayed on the displayareas of the measuring sensors.

In a further preferred embodiment of the system, the processing unitworks with an algorithm which develops further through machine learning.In a preferred embodiment of the method, an algorithm is furtherdeveloped by means of machine learning. Deep learning in particular isused as a method of machine learning. In the processing unit, a model isdeveloped once which, based on a learning algorithm, is able to learnfrom input data, i.e. in this case the image, to extract the displayareas of the measurement sensors and thus also the information depictedon them automatically.

In a training step, one image in each case is linked to an expectedresult. For example, an image from a thermochromic measuring sensor islinked to the value “52° C.”. After the training step, the model is thenused to extract the value from the image. In doing so, variations in theimage, such as lighting, other components etc. are considered, whenthere are also relevant variations that could significantly affect theimage, in the training step. In the training step, the system thenlearns to deal with the variations or to suppress the variations whenthe relevant information is extracted from the image.

The initial setup in the training step can also be supported manually bythe user manually defining the display areas in the image. Pre-trainedmodels are conceivable for the user, but also the inclusion of trainingdata for own measuring sensors by the user is possible, based on whichhe/she himself/herself trains a model for reading out. Furthermore, itis also possible for the images of the measurement sensors recorded bythe user to be collected centrally in order to train improved models forreading out. The models can be used both for partial steps, such as thelocalization of the measuring sensors in the image, the identificationof measuring sensor types, or the identification of measured values onthe display areas, as well as for an overall step from the recordedimage to the readout measured value.

The automatic image analysis of the system represents a significantadvance over the classical image processing. In the classical imageprocessing an expert must provide a special algorithm for each type ofmeasuring sensor, and possible variations in the image caused by thecamera position, other components of the microscope system, ambientlight, etc., can lead to problems. This means a great deal ofdevelopment effort even for a small number of different types ofmeasurement sensors. If additional measuring sensor types are to besupported, the algorithm must be expanded by an expert. By using machinelearning the extensibility of the system is improved. In order toinclude new types of measuring sensors, data has to be included inmachine learning and the model has to be retrained, but the design ofthe model and the learning algorithm remain unchanged. In addition, themachine learning methods significantly improve the quality of theresults compared to traditional image processing methods. Machinelearning also increases the speed, as deep learning techniques cancalculate responses in real time, even for complex tasks. In addition,continuous learning is made possible by the collection of further,improved data, without an image processing algorithm designed byexperts, which is associated with a high workload.

In a preferred embodiment of the method for state monitoring of themicroscope, a measured value is assigned to a display area manually,that is to say directly by the user, and is displayed on the displaydevice.

The invention is explained in more detail below by way of example withreference to the drawing. In the drawing:

FIG. 1 shows an illustration of a system for state monitoring of amicroscope,

FIG. 2 shows an illustration of a system for state monitoring of amicroscope with illumination, and

FIG. 3 shows a flow chart of a method for state monitoring of amicroscope.

FIG. 1 illustrates schematically a system S for state monitoring of amicroscope. Along an optical axis OA, a microscope 1 with a lens (notshown separately), captures a sample 4 located on sample carrier 2. Atemperature sensor 6 with a display area 6 a is arranged on the lens.Display area 6 a is formed as a temperature display device whichdisplays the temperature of the lens. Furthermore, a pH sensor 8 with adisplay area 8 a is attached to sample carrier 2. Display area 8 a isprovided as a color field on the pH sensor and shows a change in the pHvalue of a component, for example of sample 4, by means of color coding.Furthermore, a further temperature sensor 10 with a display area 10 a isprovided near sample 4. On the temperature sensor, display area 10 a isdepicted an external temperature display device which, for example,displays the temperature of the environment near the sample in a scaledmanner. All display areas 6 a, 8 a, 10 a are located in a field of view14 of a camera 12. Camera 12 is connected with a processing unit 16,which feeds a display device 18.

Temperature sensor 6 captures at least one time-variable chemical and/orphysical quantity. In the case of using temperature sensor 6, as is thecase in FIG. 1 , the time-variable chemical and/or physical quantity isthe temperature of the lens. The measuring sensor can be formed as athermochromic element which changes its color on display area 6 a or hasa temperature display 6 a on which a measured value is displayeddirectly. Other ways of visualizing the measurement of the time-variablechemical and/or physical quantity are also conceivable. It is importantin this case only that the outer appearance of the display area of themeasuring sensor permits to draw conclusions about the captured measuredvalue, in the case of temperature sensor 6, about the temperature.

pH sensor 8 is attached to sample carrier 2. Said pH sensor 8 is used tomeasure a time-variable chemical and/or physical quantity on the samplecarrier, in this case the pH value. In this case it is also possible toform the measuring sensor as a thermochromic element. Another displayarea 8 a may be provided for visualization of the measurement result, sothat the display area of the measuring sensor permits to drawconclusions about a measured value, in this case, the pH value. Codingbased on a color field is possible here as a display area, for example.Furthermore, a temperature sensor 10 is provided in the system accordingto FIG. 1 . Said temperature sensor 10 also has a display area 10 a thatdisplays the measured value. In this case, it is formed as a temperaturedisplay 10 a externally to temperature sensor 10.

All three measuring sensors 6, 8, 10 are not connected to the processingunit 16 for the exchange of measured values. They can be formed, forexample, as passive sensors or as modules with an internal energysupply, such as battery-operated modules or modules with rechargeablebatteries. Display areas 6 a, 8 a, 10 a of all the measuring sensors 6,8, 10 are located within field of view 14 of camera 12. Camera 12 isformed and positioned suitably for this purpose. Monitoring of themeasuring sensors 6, 8, 10 is ensured continuously or at specific times.

An overview camera attached specifically to monitor display areas 6 a, 8a, 10 a, but also a camera that is already otherwise present on thesystem, can be used as camera 12. In this case said camera is aligned insuch a way that all display areas 6 a, 8 a, 10 a are located withinfield of view 14. The type and nature of the camera is irrelevant inthis case as long as the display areas to be analyzed are visible in theimages.

Camera 12 is connected to the processing unit 16 for the exchange ofimage data. The camera records images of the display areas 6 a, 8 a, 10a of the measuring sensors 6, 8, 10 continuously or at certain times andtransmits them to processing unit 16. In addition to display areas 6 a,8 a, 10 a, there are also other components that are irrelevant formonitoring in each image. Processing unit 16 evaluates each recordedimage by means of image analysis in order to capture the valuesdisplayed on display areas 6 a, 8 a, 10 a of measuring sensors 6, 8, 10.In exemplary embodiments, this includes an image analysis with thefollowing steps: finding the display areas 6 a, 8 a, 10 a of themeasuring sensors 6, 8, 10 in the image, optionally identifying the typeof measuring sensor (manufacturer, model, etc.) and then identifying themeasured value for each measuring sensor 6, 8, 10. The values aredisplayed on display device 18. In exemplary embodiments, display areas6 a, 8 a, 10 a can be read out by processing unit 16 by means of textand/or image analysis. In modifications, display areas 6 a, 8 a, 10 acontained in the image are extracted from the image and joined togetherto form an overall image in a stitching process. The overall image thenconsists of display areas 6 a, 8 a, 10 a lined up next to one another,wherein all of the components that are not relevant for monitoring, butwhich are contained in the image, are not included in the overall image.The overall image is then displayed on the display device 18.

As FIG. 1 , FIG. 2 illustrates the system S for state monitoring of amicroscope schematically. It addition to the system of FIG. 1 it has anillumination source 20, which produces an illumination field 22. Displayareas 6 a, 8 a, 10 a of measuring sensors 6, 8, 10 are activelyilluminated with illumination source 20 in order to be independent ofambient light, for example.

FIG. 3 illustrates a flow chart of the method for state monitoring of amicroscope. It consists of steps S1 to S5.

In a step S1, measuring sensors 6, 8, 10 capture at least onetime-variable chemical and/or physical quantity of a microscope. Eachmeasuring sensor 6, 8, 10 has a display area 6 a, 8 a, 10 a. Thetime-variable chemical and/or physical quantities can be, for example,the temperature, the CO2 content, the O2 content, the pressure, the pHvalue, the humidity, the exposure to light and/or the filling level. Ina step S2, the measured, time-variable chemical and/or physicalquantities are displayed on display areas 6 a, 8 a, 10 a of measuringsensors 6, 8, 10. The image is recorded either continuously or atspecific times in a step S3. In a step S4, the image is evaluated byprocessing unit 16. For this purpose, processing unit 16 preferablyworks with automatic image analysis, wherein first display areas 6 a, 8a, 10 a are found in field of view 14 and identified locally and then,optionally, the type of measuring sensor (manufacturer, model, etc.) isidentified before the displayed measured value is identified for eachidentified display area 6 a, 8 a, 10 a. The results of the automaticimage analysis in step S4 are finally displayed on display device 18 ina step S5.

In modifications, display device 18 is formed as a warning device.Measured values are defined, the overshooting or undershooting of whichis identified by processing unit which then warns the user. For thispurpose, display device 18 is activated by processing unit 16, whichprovides a warning signal to the user.

In modifications, processing unit 16 is connected to a memory. Themeasured values are provided with a time stamp by processing unit 16 andstored in the memory. The stored measured values can then be retrievedby processing unit 16 at any time.

In a further exemplary embodiment, it is possible for the user tospecify a measured value of display areas 6 a, 8 a, 10 a manually, whichis displayed on display device 18.

Several possibilities have been described by means of which processingunit 16 can read out the measured values from display areas 6 a, 8 a, 10a and extract them from the image. Processing unit 16 works with analgorithm when carrying out the extraction. In modifications thisalgorithm is further developed using machine learning. Deep learning inparticular is used as a method of machine learning. In the processingunit, a model is developed once which, based on a learning algorithm, isable to learn from input data, i.e. the image in this case, to extractdisplay areas 6 a, 8 a, 10 a of measuring sensors 6, 8, 10 and thus alsothe information displayed on it automatically.

LIST OF REFERENCE NUMERALS

-   -   1 Microscope    -   2 Sample carrier    -   4 Sample    -   6 Temperature sensor    -   6 a Display area    -   8 pH sensor    -   8 a Display area    -   10 Temperature sensor    -   10 a Display area    -   12 Camera    -   14 Field of view    -   16 Processing unit    -   18 Display device    -   20 Light source    -   22 Illumination field    -   OA Optical axis    -   S1 Step 1    -   S2 Step 2    -   S3 Step 3    -   S4 Step 4    -   S5 Step 5

The invention claimed is:
 1. A system for state monitoring of amicroscope, the system having: at least one measuring sensor in eachcase for capturing at least one time-variable chemical and/or physicalquantity, a camera for recording an image in a field of view, and aprocessing unit, wherein: the at least one measuring sensor has adisplay area and displays a measured value thereon for the at least onecaptured time-variable chemical and/or physical quantity, the camera isarranged so that the display area of the at least one measuring sensoris located in the field of view, the processing unit is configured toevaluate the image and to extract therefrom the measured valuesdisplayed on the display area contained in the image, and the processingunit is configured to: (i) evaluate the image and to extract therefrominformation identifying a type of the at least one measuring sensor,(ii) evaluate the image to independently determine the location of thedisplay area of the at least one measuring sensor based on machinelearning, or (iii) a combination of (i) and (ii).
 2. The systemaccording to claim 1, wherein the at least one measuring sensor is notconnected with the processing unit for the exchange of measured values,and the camera is a component of the microscope.
 3. The system accordingto claim 1, wherein the processing unit is configured to filter out thedisplay area contained in the image and to join them together to form anoverall image.
 4. The system according to claim 1, wherein theprocessing unit is configured to read out the measured values from thedisplay area by means of text recognition and/or image analysis.
 5. Thesystem according to claim 1, wherein the processing unit has a memory inwhich the measured values provided with a time stamp are stored.
 6. Thesystem according to claim 1, wherein a display device is provided whichdisplays the results of the extraction of the measured values displayedin the display area.
 7. The system according to claim 6, wherein thedisplay device is formed as a warning device by means of which thesystem, controlled by the processing unit, signals the overshooting orundershooting of fixed measured values to the user.
 8. The systemaccording to claim 1, wherein the results of the extraction of themeasured values displayed in the display area are transmitted to acontrol loop and/or a workflow.
 9. The system according to claim 1,wherein the at least one measuring sensor is formed as a passive sensoror as a module with internal power supply.
 10. The system according toclaim 1, wherein an illumination source is provided, which illuminatesthe display area of the at least one measuring sensor.
 11. A method forstate monitoring of a microscope, the method comprising: providing atleast one measuring sensor, which in each case captures at least onetime-variable chemical and/or physical quantity, recording an image witha camera, wherein: the at least one measuring sensor has a display areaand a measured value is displayed thereon for the at least onetime-variable chemical and/or physical quantity, the image is recordedso that it contains the display area of the at least one measuringsensor, the display areas are identified in the image, evaluating theimage to extract therefrom the measured values displayed on the displayarea contained in the image, and performing one or both of (i)evaluating the image and extracting therefrom information identifying atype of the at least one measuring sensor or (ii) evaluating the imageto independently determine the location of the display area of the atleast one measuring sensor based on machine learning.
 12. The methodaccording to claim 11, wherein the at least one measuring sensor is notconnected with the processing unit for the exchange of measured values,and the camera is a component of the microscope.
 13. The methodaccording to claim 11, wherein the display area contained in the imageare filtered out and joined together to form an overall image.
 14. Themethod according to claim 11, wherein the measured values are read outfrom the display area by means of text recognition and/or imageanalysis.
 15. The method according to claim 11, wherein the measuredvalues are provided with a time stamp and stored in a memory.
 16. Themethod according to claim 11, wherein the results of the extraction ofthe measured values displayed in the display area are displayed with adisplay device.
 17. The method according to claim 16, wherein anovershooting or undershooting of fixed measured values is signaled tothe user by the display device formed as a warning system.
 18. Themethod according to claim 11, wherein the results of the extraction ofthe measured values displayed in the display area are used in a controlloop and/or a workflow.
 19. The method according to claim 11, whereinthe display area of the at least one measuring sensor is illuminatedwith an illumination source.
 20. The method according to claim 11,wherein a measured value is manually assigned to an optical ormechanical component of the microscope and this measured value isdisplayed on the display device.